Spine = 0.5183 in.

Refining

Design

for Business

USING ANALYTICS, MARKETING, AND TECHNOLOGY

TO INFORM CUSTOMER-CENTRIC DESIGN

MICHAEL KRYPEL

Refining

Design

for Business

Using analytics, marketing, and technology

to inform cUstomer-centric design

Michael KRypel

refining design for Business:

Using analytics, marketing, and technology to inform customer-centric design

michael krypel

This Adobe Press book is published by Peachpit.

For information on Adobe Press books and other products, contact:

Peachpit

www.peachpit.com

For the latest on Adobe Press books, go to www.adobepress.com

To report errors, please send a note to errata@peachpit.com

Peachpit is a division of Pearson Education.

Copyright © 2014 by Michael Krypel

Project Editor: Valerie Witte

Production Editor: Becky Winter

Developmental and Copyeditor: Anne Marie Walker

Proofreader: Liz Welch

Composition: Danielle Foster

Indexer: James Minkin

Cover and Interior Design: Mimi Heft

Cover and Interior Illustrations: Paul Mavrides

notice of rights

All rights reserved. No part of this book may be reproduced or transmitted in any form by any means, electronic, mechanical, photocopying, recording, or otherwise, without the prior written permission of the publisher. For information on getting permission for reprints and excerpts, contact permissions@

peachpit.com.

notice of liability

The information in this book is distributed on an “As Is” basis, without warranty. While every precaution has been taken in the preparation of the book, neither the author nor Peachpit shall have any liability to any person or entity with respect to any loss or damage caused or alleged to be caused directly or indirectly by the instructions contained in this book or by the computer software and hardware products described in it.

Any views or opinions presented in the interviews in this book are solely those of the author and interviewee and do not necessarily represent those of the companies included in this book.

trademarks

Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in this book, and Peachpit was aware of a trademark claim, the designations appear as requested by the owner of the trademark. All other product names and services identified throughout this book are used in editorial fashion only and for the benefit of such companies with no intention of infringement of the trademark. No such use, or the use of any trade name, is intended to convey endorsement or other affiliation with this book.

ISBN-13: 978-0-321-94088-9

ISBN-10: 0-321-94088-1

9 8 7 6 5 4 3 2 1

Printed and bound in the United States of America

To Darren Johnson, Jennifer Sun, Kripa Nerlikar,

Ramona Meyer-Piagentini, and Stephen Ratpojanakul:

Thank you for your guidance and enthusiasm over the last eight years,

and for your tremendous contributions to building the optimization field.

Acknowledgments

I will be forever grateful to all the wonderful people from Adobe, Omniture, and

Offermatica for allowing me to be part of their exceptional teams, including Aaron

Graham, Adam Fayne, Adam Justis, Adam Wood, Alan Gurock, Amy Lam, Andre

Prevot, Ann Chen, Aseem Chandra, Barbara Dawson, Bianca Slade, Bill Ozinga, Bill

Peabody, Brad Kay, Brent Dykes, Brian Hawkins, Brian Ivanovick, Brittany Chandler,

Cameron Barnes, Christi Terjesen, Christine Yarrow, Christy Armstrong, Colin Lewis,

Colleen Nagle, Daniel Hopkins, Daniel Wright, Darren Johnson, Darrin Poole, David

Baker, David Hoye, David Humphrey, Debra Adams, Derek Bryce, Don Abshire, Doni

Lillis, Doug Mumford, Drew Burns, Drew Phillips, Eddie Ramirez, Ehren Hozumi,

Garrett Ilg, Gene Holcombe, Georgia Frailey, Heather Razukas, Hiro Awanohara, Jacob

Favre, James Roche, Jameson O’Guinn, Jamie Stone, Jason Haddock, Jason Hickey,

Jason Holmes, Jeff Fuhriman, Jennifer Sun, Jim Sink, John Kucera, John Mosbaugh,

Jonathan Mendez, Jonathan Weissbard, Justin Patrick, Ka Swan Teo, Kaela Cusack,

Katie Cozby, Kellie Snyder, Kendra Jenkins, Kevin Lindsay, Kevin Scally, Kevin Smith,

Kripa Nerlikar, Kyle Ellis, Kyle Johnson, Lacey Bell, Lambert Walsh, Lance Jones, Lily

Chiu, Liz Quinn, Mandeep Sidhu, Mark Boothe, Matthew Lowden, Matthew Roche,

Matthew Smedley, Matthew Thurber, Michael Curry, Michael Evensen, Mikel Chertudi,

Neha Gupta, Norman Dabney, Paige Burton, Peter Callahan, Rachel Elkington, Rameen

Taheri, Ramona Piagentini, Rand Blair, Reuben Poon, Richard Oto, Rob Cantave, Ron

Breger, Rotem Ben-Israel, Russell Lewis, Sachie Reichbach, Sarah Ferrick, Serge St. Felix, Shoaib Alam, Stephen Frieder, Stephen Ratpojanakul, Steve O’Neil, Thejas Varier, Tiffany Olejnik, Tom Ratcliff, Tony DiLoreto, Tracy Harvey, Vincent Cortese, Vladimir Sanchez

Olivares, Wallace Rutherford, Whitney Littlewood, and Zoltan Liu.

I am extremely thankful to everyone who agreed to be interviewed or who helped

connect me with people to interview, including Justin Ramers from Active Network;

Andrew Switzer from Ally Bank; Tom Lau from American Express®; Kevin Gallagher

and Thomas Gage from AutoTrader; Adam Crutchfield from Axcess Financial; Brandon

Proctor and Justin Bergson from Build.com; Chris Kahle from Caesars; Joanne Pugh and

Stephanie Paulson from Central Restaurant Products; Kyle Power from CHG Healthcare;

Christine Cox, John Williamson, Jonathan Stein, and Ruth Zinder from Comcast; Ed Wu,

Emily Campbell, Étienne Cox, Isabelle Mouli-Castillo, Joel Wright, Lester Saucier, Nazli Yuzak, and Will Close from Dell; Sandy Martin from Dollar Thrifty; Zimran Ahmed

from Electronic Arts; Nate Bolt and Slater Tow from Facebook; Thomas Jankowski from

FlightNetwork; Karina van Schaardenburg and Simon Favreau-Lessard from Foursquare;

Jerome Doran, Jon Wiley, and Krisztina Radosavljevic-Szilagyi from Google; Linda Tai

from Hightail; Phil Corbett from IBM; Ajit Sivadasan, Ashish Braganza, Lewis Broadnax,

and Siping Roussin from Lenovo; Amy Parnell and Lea Ann Hutter from LinkedIn; Pete

Maher from the Luma Institute; Kenyon Rogers from Marriott; Peter Davio and Steven

Webster from Microsoft; Kyle Rush from Obama for America; Blake Brossman, Natalie

Bonacasa, and Ujjwal Dhoot from PetCareRx; Amit Gupta from Photojojo; John Pace

from RealNetworks; Chris Krohn, Phil Volini, and Sarah Nelson from Restaurant.com;

Matt Curtis and Roger Scholl from Saks Fifth Avenue; Ryan Pizzuto from T-Mobile;

Eileen Krill and Mary Bannon from The Washington Post; and Matthew Pereira and Rob

Blakeley from WebMD.

Thank you to Kelly Patterson for her fabulous editing, to Rosemary Knes for her careful

proofing, and to Jeff Patterson for introducing me to them. Thank you also to everyone

at the Adobe Press and Pearson Education for their hard work, including Anne Marie

Walker, Becky Winter, Damon Hampson, Danielle Foster, Jim LeValley, Liz Welch, Mimi

Heft, Ted Waitt, Valerie Witte, Victor Gavenda, and Vidya Subramanian Ravi.

Thank you to my mom, Merrill Janover, whose curiosity, creativity, and love of learning

are an inspiration, and to my brothers, David Krypel and Brian Krypel, for their

encouragement and comfort. Thank you to Alan Schorn, Bob Klein, Chayym Zeldis,

David Roth, Elizabeth Metz, Jarek Koniusz, Jeffrey Gilden, Jerry Denzer, John Cave, John Zannos, Joseph Rutkowski, Nina Zeldis, Philip Sorgen, Robert Abrams, and all my other

wonderful teachers for their guidance, confidence, and inspiration. Thank you also to

Daniel Schweitzer, Bonnie Walters, Greg Lowder, and Scott Epstein for their feedback

on drafts of the book and for their amazing support.

Contents at a Glance

Part 1 creating engaging customer experiences

1

chaPter 1 The art and Science of Business . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3

chaPter 2 Using Data to inform Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

Part 2 the iterative optimization methodology

39

chaPter 3 Methodology Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

chaPter 4 Qualitative Research . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65

chaPter 5 Quantitative Research and Optimization Roadmap . . . . . . . . . . . . . 87

chaPter 6 Optimization in action . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .107

chaPter 7 Building an Optimization Organization . . . . . . . . . . . . . . . . . . . . . . . . .129

Part 3 Visual Business cases

141

index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 279

Contents

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiv

The Age of Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiv

How This Book Is Structured . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .xv

Part 1 creating engaging customer experiences

1

chaPter 1 The art and Science of Business . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3

The Importance of Design to Business . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4

Customer-Centric Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6

Designed to Evolve:

The Iterative Optimization Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . .7

Visual Business Cases. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10

The Washington Post: Social Networking Links . . . . . . . . . . . . . . . . . . . .11

2012 Presidential Election: Dinner with Barack Image Test . . . . . . . .20

Building an Optimization Organization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .22

The Skill Set of the Future . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .23

Up Next . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .23

References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .24

chaPter 2 Using Data to inform Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

The Traditional Design Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .26

Design Decisions Based on Bad Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .26

Design Decisions Based on Pixel Politics . . . . . . . . . . . . . . . . . . . . . . . . . . .27

Reexamining the Design Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .28

Design Decisions Based on Good Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . .29

The Cultural Benefits of Testing Designs . . . . . . . . . . . . . . . . . . . . . . . . . . .31

Online Design Is Not Fine Art . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .31

Advice from Business Leaders . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .34

Up Next . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .36

References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .36

x

Refining DeSign fOR BUSineSS

Part 2 the iterative optimization methodology

39

chaPter 3 Methodology Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

Step by Step . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .42

The Paper Trail: Why It Matters and How to Do It . . . . . . . . . . . . . . . . .43

The Optimization Roadmap . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .44

The Optimization Plan . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .45

Wireframes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .45

Design Comps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .45

Optimization Launch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .46

Optimization Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .46

Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .46

Communication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .47

Case Study: PetCareRx . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .47

Optimization Roadmap . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .47

Optimization Plan . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .51

Wireframes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .52

Design Comps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .55

Optimization Launch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .55

Optimization Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .61

Up Next . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .63

chaPter 4 Qualitative Research . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65

Benefits of Qualitative Research . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .66

Learning More About Customer Goals. . . . . . . . . . . . . . . . . . . . . . . . . . . . .66

Starting to Answer Qualitative Questions . . . . . . . . . . . . . . . . . . . . . . . . .67

User Experience Reviews . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .68

Online Customer Behavior . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .68

Experiential Business Review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .69

Experiential Competitive Review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .69

Documenting Experiential Reviews . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .70

Don’t Act Too Quickly . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .72

Observational Customer Research . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .72

Other Research Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .75

Ethnographic Studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .75

Surveys . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .77

Customer Panels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .77

Diary Studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .78

Card-sorting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .78

Feedback Forms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .78

Acting on Qualitative Research . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .79

contents

xi

Case Study: Comcast . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .79

Take a Step Back . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .85

Up Next . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .86

References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .86

chaPter 5 Quantitative Research and Optimization Roadmap . . . . . . . . . . . . . 87

Benefits of Quantitative Studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .88

Starting with Quantitative Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .88

Telling a Story Using Insights and Impact . . . . . . . . . . . . . . . . . . . . . . . . .89

Online Success Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .90

Analytics in Action: A Case Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .93

Action Plan: Roadmap Assembly . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .102

Prioritizing Where to Test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .102

Prioritizing Test Ideas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .103

Up Next . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .106

References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .106

chaPter 6 Optimization in action . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .107

Optimization Plan . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .108

Test Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .108

Number of Recipes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .109

Test Type . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .114

Success Metrics and Segments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .117

Wireframes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .119

Structure of a Wireframe . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .119

Assembling Wireframes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .119

Creating Great Wireframes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120

Design Comps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122

Optimization Launch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123

Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123

Quality Assurance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124

Optimization Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125

Communication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126

Up Next . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127

References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127

chaPter 7 Building an Optimization Organization . . . . . . . . . . . . . . . . . . . . . . . . .129

Optimization Maturity Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130

Maturity Model Checklist . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130

Core Optimization Team . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .132

xii

Refining DeSign fOR BUSineSS

Hiring and Nurturing Talent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133

Roles and Responsibilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133

Core Optimization Team Roles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135

Hiring for Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .137

Up Next . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140

Part 3 Visual Business cases

141

Adobe . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .143

Visual Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .143

Ally Bank . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146

Visual Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .147

American Express® . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150

Visual Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .151

Caesars Entertainment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153

Visual Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154

Comcast . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158

Visual Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160

Dell . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166

Visual Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .168

Dollar Thrifty Automotive Group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .172

Visual Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .173

Electronic Arts. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .177

Visual Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .179

Facebook . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185

Visual Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186

Foursquare . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .191

Visual Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194

Google . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196

Visual Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 204

Hightail . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .210

Visual Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .211

IBM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .213

Visual Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .215

LinkedIn . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 220

Visual Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223

Marriott International . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 226

Visual Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 227

Obama for America . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .231

Visual Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 236

Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 249

contents

xiii

PetCareRx . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 251

Visual Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 252

Saks Fifth Avenue . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 258

Visual Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 258

T-Mobile . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 262

Visual Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263

The Washington Post . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 268

Visual Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .271

index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 279

Introduction

The age of optimization

This book aims to help businesspeople apply a comprehensive and powerful new way of

building business and customer value online, using a proven methodology that works

for any size of business, in any industry. Highly interdisciplinary in nature, the Iterative Optimization Methodology will be especially useful to:

C

C

Business executives. Aiming to meet online customer needs through a rigorous

research and evaluation process that also fosters creativity, the optimization method

is appealing to business leaders of all ranks, including C-Level executives.

C

C

Marketers, designers, and information architects. Great marketers and designers

were among the first professional groups to embrace the customer-centric design

techniques described in this book.

C

C

Data analysts. Data analysts are essential to the success of almost every step of the Iterative Optimization Methodology. They provide users throughout the business

with access to crucial data and help them to act on analytics-based insights.

C

C

Engineers and solutions architects. Engineers and solutions architects are often the unsung heroes of online business. They spend much of their time setting up systems to deploy designs without knowing whether what they’re working on is helping

the company and its customers. The optimization method allows them to find out

whether or not their efforts have been fruitful.

C

C

Students, including MBA candidates. Students can learn more about an exciting

new field—one in which there is high demand for people who excel at creating great

wireframes, running optimization programs, and other skills described in this book.

how This Book is structured

This book is divided into three parts:

Part 1: Creating Engaging Customer Experiences. This part discusses the level of importance design now plays in the business world, challenges the standard design

process implemented by most companies, and introduces the Iterative Optimization

Methodology by showing how design testing can lead to more creative and

impactful designs.

Part 2: The Iterative Optimization Methodology. Using real-life examples, this part describes how to drive business and customer value in step-by-step detail. It shows

how companies can integrate qualitative and quantitative customer research, prioritize

website sections and design ideas for testing, experiment with new designs under real

market conditions, and scale optimization techniques across their organization.

Part 3: Visual Business Cases. In this part, business leaders from 20 companies, including Google, Facebook, Comcast, Marriott, and American Express®, share examples

of their favorite design tests and discuss practical approaches for using data to inform customer-centric design.

This page intentionally left blank

This page intentionally left blank

Chapter 4

Qualitative research

Many businesspeople, even those responsible for the online

experience, don’t spend nearly enough time trying to see their

business through the eyes of the customer. Qualitative research offers

an invaluable way not only to get into the customer mind-set, but also

to come up with ideas for innovation. This chapter discusses several

types of qualitative research and shows how they can be used to

generate test ideas while providing insights that will help guide the

business in an evolving marketplace.

66

Refining Design foR Business

Benefits of Qualitative research

Qualitative research is the art of capturing the customer’s perspective on the online experience offered by a business and its competitors. Companies often neglect to do qualita-

tive research and instead take the easy route by relying solely on analytics software to understand their customers’ online behavior. However, quantitative research can only

reveal how site users are acting; it can shed very little light on what they are thinking or feeling. In short, it is no substitute for truly seeing a company through the eyes of its users.

This is not always a comfortable experience for business leaders, but it is key to helping them understand the customer perspective, adapt their business to evolving conditions,

and generate ideas for new designs—tasks that metrics alone cannot fulfil .

In particular, qualitative research is critical to identifying:

C

C

Unmet customer needs. If a business is failing to provide customers with what

they’re looking for, no analytics report can shed light on that gap. By watching and

listening to customers, or by re-creating their online experience firsthand, research-

ers can discover users’ expectations, unmet needs, and any sources of frustration.

C

C

The why behind the analytics data. Sometimes metrics can generate more questions than answers. For example, an analytics report showing an increase in page

views per visitor isn’t always a good sign. Although it may indicate customers’ satis-

faction through deep engagement with the site, it may also indicate confusion about

how to add products to their cart, or reveal unsuccessful attempts to locate informa-

tion such as shipping rates or to find pages on the site that customers think should

exist but do not.

Most qualitative research methods work with small, even statistically insignificant, sample sizes. As a result, this kind of research is best used for understanding broad ques-

tions—such as whether customers understand the basic business offering and whether

it’s something they need—as opposed to informing decisions about how to design every

detail of a business. For the same reason, researchers often verify hypotheses based on

qualitative data by referencing web analytics or testing to see if the hypotheses apply to a large number of customers.

Learning More About Customer Goals

The first step in qualitative research is to define realistic customer goals that explain why customers are choosing to interact with a business. This is not as easy as it sounds: Customers may access a site for very different reasons, and their goals can and do change

over time. As Rob Blakeley, Senior Product Manager for WebMD, put it, “Our customers’

goals are as different as there are blades of grass in New York City’s Central Park.”

However, most businesses can easily identify a preliminary list of the most likely customer goals and adapt this list as researchers gather further information. For example,

ChapteR 4: QuALitAtive ReseARCh

67

a financial services company’s customer goals may include opening or closing a deposit

account, sending and depositing checks from a mobile device, getting a replacement

credit card while traveling abroad, and so on.

If a businessperson is struggling with this task, it may help to ask the question: If customers could come to the business looking for help with only one thing, what would it be?

It’s much better to create a top-notch experience that can satisfy one important goal

shared by many customers than to create several mediocre experiences in an attempt

to address many different objectives.

starting to Answer Qualitative Questions

The next step in qualitative research is to build a list of questions to explore. Like the customer goals, this list is preliminary and will change over time. The following is a list of potential questions that companies may use as a starting point in this process; of

course, each business will tailor the list to its own priorities.

Fundamental questions:

C

C

Why do customers come to the business?

C

C

Why do they leave?

C

C

Do customers understand what the business has to offer?

C

C

Do customers want what the business has to offer?

C

C

Is there anything customers want from the business that it is not providing?

C

C

When analytics data shows areas of concern—for example, high drop-off rates,

repeat page views, and so on—what are the reasons for the customers’ actions?

C

C

Which product or service is most important to customers?

C

C

If the business could focus on only one product or service, what should it be?

On finding products, services, and content:

C

C

Is the full breadth and depth of the product/services/content offering apparent?

C

C

Are there too many or too few products/services/content to choose from?

C

C

Do customers easily find products/services/content using site navigation

and/or search?

On the customer’s decision-making process:

C

C

Are products/services/content easy to differentiate from one another, or are they

too similar, leading to customer confusion over which to choose?

C

C

Are the cross-sells, up-sells, and promotions clear? Do they interfere with the

primary purchase?

C

C

Is there enough information to make a purchase decision?

68

Refining Design foR Business

C

C

Is it clear how to find related or popular content? Does it interfere with the primary

content experience?

On conversion:

C

C

Are there barriers to easily checking out?

C

C

Are prices, taxes, shipping, returns, and security protections apparent? Are any of

them barriers to checking out?

C

C

Are there barriers to easily viewing additional content?

On the competition:

C

C

Which competitors do customers go to, and how do the preceding questions apply

to these competitors?

C

C

Do competitors offer a product or service that would complement the business’s

offerings, such as accessories for a product the business sells? If so, should the busi-

ness offer it as well?

Note that some of the preceding questions are based on quantitative research; however, it is not essential to have performed quantitative studies before tackling qualitative research.

Most businesses already have a basic understanding of their data, and the iterative

method means new insights, from both kinds of studies, will be added to the mix through-

out the process, al owing team members to refine and add to their list of questions.

The next step in qualitative research is to try to answer these questions through first-

hand experience, observing customers, and other methods, as described in the follow-

ing sections.

User experience reviews

In user experience reviews, or heuristic reviews, researchers strive to experience the business firsthand in the same way customers do; then they do the same on competitors’

websites for comparison. Before we examine user experience reviews in detail, it is helpful to explore some guidelines on customer behavior that can help the researcher, or

reviewer, make their customer experience as authentic as possible.

Online Customer Behavior

Customers generally display the following tendencies when interacting with an online

business:

C

C

Customers are goal-oriented. They’re looking for a place to click on the page or

app that will take them to the specific product, service, or content that constitutes

their objective.

ChapteR 4: QuALitAtive ReseARCh

69

C

C

Customers scan pages quickly. They skip over details until they’ve found what

they’re looking for. For example, they may click through a business’s homepage and

several category and product pages rapidly, navigating with the help of images or

keywords until they find a product they’re interested in; only then are they likely to

read details such as the price, feature list, reviews, and shipping policy. They are very good at ignoring anything that looks like advertising, a tendency that is sometimes

called banner blindness.

C

C

Customers don’t act like professionals. They don’t think about whether the navi-

gation is consistent, which page they’re on within the site, how individual page ele-

ments are working, what the business is trying to get them to do next, and so forth.

experiential Business Review

Researchers first choose a realistic customer goal from the prepared list, and then follow a “customer scenario”—the actions a customer would take to pursue that objective. It is

important here to step out of the professional mind-set and just experience the website

as a customer would: If the reviewer can relate personally to the goal, all the better. For example, for a retail business the reviewers could go through the “browse and purchase”

process for an item they’re interested in buying or would like to get as a birthday gift for a friend; for a travel business, the reviewers could look into booking their next trip.

Taking notes along the way, reviewers should strive to do anything customers would

do—call customer support, abandon the site when unable to attain their goal easily, and

so on. If many customers visit the site from mobile devices, reviewers should try going

through the scenarios using a phone or tablet.

After experiencing one scenario, the reviewers retrace each step of the experience to

critique it as a professional, noting any opportunities for improvement. Was it easy

to accomplish the goal? Were some elements especially difficult to navigate? Were

there steps that confused the reviewers? The reviewers then repeat this process for the

remaining customer goals.

experiential Competitive Review

Because customers are also visiting competitors’ websites, reviewers should do so, too.

They follow the same procedure as before, exploring the same scenarios as customers,

and then return to each step to critique the experience as professionals. To make the

experience especially organic, reviewers may even start on their own business’s site and then abandon it for a competing site based on their natural browsing behavior or on

analytics data that shows where customers tend to abandon.

70

Refining Design foR Business

Documenting experiential Reviews

It’s important to document the heuristic reviews to share the findings and to prepare

for the observation of real customer interactions. The documentation process should

consist of screen shots showing the step-by-step interaction path, notes explaining

the screen shots, and an updated list of qualitative questions based on the review.

For example, researchers may ask whether customers also struggle or get confused

where the reviewers did or whether users find it easy to navigate the same paths as

the reviewers.

The following example illustrates how this process typically unfolds through a user

experience review of the Gap, Inc. conducted in 2013. Figure 4.1 encapsulates the main steps of the experiential review.

The accompanying notes for the steps in the figure include:

1. Reviewer visits Gap.com to look for a summer dress.

2. Reviewer sees red dress in middle right that looks cute and clicks on it. Reviewer is not taken to the dress but to a page listing many dresses that don’t look like the one

she clicked.

3. Reviewer sees a similar dress in a different color on the second row and clicks on it.

Reviewer decides to purchase it. She tries to change the color to red by clicking on

the large image of the dress in the middle of the screen, but doing so simply shows

a zoomed-in view of the dress.

4. Just before abandoning, reviewer clicks on the small image of the dress in the middle of the screen.

5. Reviewer now sees the color palette and realizes that the dress is no longer available in red. She decides to buy it in coral as pictured. She clicks on “add to bag” but noth-ing happens, so she leaves to shop at a competitor’s site. (The “add to bag” link failed because she first needed to select a size; there was no error message explaining this.)

Upon further critiquing the experience, the following questions were generated for the

observational stage of the research:

CC

Do customers also struggle to find the product page for an item they liked on the

homepage?

CC

Even though it was jarring to be taken to a page with hundreds of dresses after click-

ing on a single one, would it be a good customer experience to arrive on a page like

that after clicking on the “Dresses” category in the Women’s section of the site?

CC

Do customers also struggle to figure out what colors an item comes in and how to

choose a different color?

CC

Do customers also find it difficult to add an item to their cart?

ChapteR 4: QuALitAtive ReseARCh

71

1

3

4

2

5

Figure 4.1 It’s easy for customers to get confused or frustrated

online. In this example of a user experience review, the customer

would have abandoned the site after struggling to purchase the

red dress she liked on the homepage.

72

Refining Design foR Business

C

C

Do customers also try to buy before selecting a size and get confused when no

message reminds them to pick a size or offers to help them do so?

Researchers seek to validate these questions by observing customers interacting with

the site and by digging into the site analytics. Team members then create hypotheses,

or proposed answers to these questions, and attempt to vet the hypotheses through test

ideas, which are added to the Optimization Roadmap.

Don’t Act too Quickly

The purpose of the user experience review is to gain insights into the customers’ experiences by stepping into their shoes for a moment. However, team members should resist

making immediate changes to the site based solely on heuristic reviews unless clear

technical errors are uncovered. Otherwise, the changes may reflect the perspective of

online professionals, not the typical customer.

Conducting realistic user experience reviews is a skill that can be learned over time,

but improvement requires an adequate feedback loop: Reviewers need to watch actual

customers interact with the business and test new designs, not simply criticize the

company’s site in a vacuum.

Researchers must also exercise caution when exploring competitors’ sites. All too often

businesses assume a competitor’s designs are effective and simply copy them without

finding out whether these designs would work for their own business. As a result, not

only do they risk adopting ineffective designs, but their sites also wind up looking very similar to those of their competitors.

One way to avoid this pitfall is to review sites outside of the business’s circle of competitors when looking for new ideas; no matter where they find new concepts, companies

must test them before pushing them live to all customers.

Observational Customer research

Directly experiencing a site can be a great way to generate questions that lead to test

ideas, but it’s no substitute for spending time with real customers. As noted in Chapter 1, users aren’t very good at articulating exactly what they need, so rather than asking

customers for their opinions, observational research instead pays close attention to

their behavior as a way of understanding their perspective. Team members strive to

watch customers interacting naturally with the site; they provide them with opportuni-

ties to give open-ended feedback and avoid guiding them too much or prompting them

with leading questions.

ChapteR 4: QuALitAtive ReseARCh

73

Customers are recruited in a variety of ways; although some volunteer, they’re more

often compensated through some form of payment, such as a check or gift card. Even

though customers know they’re being observed, the goal of this type of study is to create as natural a setting as possible, so it may take place not only in research facilities, but also in cafés, customer homes, and other venues. Sessions are conducted one-on-one,

with additional researchers observing behind one-way glass or through a video feed.

Depending on the business, sessions can last anywhere from 10 to 60 minutes. High-

level trends usually start to emerge after watching four or five customers, but to be on the safe side, researchers often observe eight to ten customers during each study. The

frequency of observational studies varies according to the company, but most busi-

nesses should conduct them at least quarterly. Many sizable and successful companies

conduct this type of research whenever they have pressing unanswered questions about

their business, which could arise several times per month.

The following list outlines the basic framework for this type of research, as well as some best practices for observing customers online:

Recruitment

C

C

The first few times a business conducts this type of research, it may be helpful to go

through a recruitment agency and use an expert moderator to conduct the sessions.

Later, the business can recruit by placing an ad online on a site like Craigslist or

work with existing customers contacted through an internal email list.

C

C

It is useful to recruit customers based on an experience they recently had or an

upcoming experience that the business can help with. For example, a hospitality

business might recruit customers who have recently booked a hotel room online or

who need to book a room for an upcoming trip.

C

C

If possible, the customers should not know the name of the business, because this

information might influence their behavior. For example, recruiters can tell them the

study is about online user behavior related to the general category of tasks the busi-

ness is interested in observing (purchasing clothing, reading the news, searching for

general information, and so on).

Setting scenarios

C

C

When customers arrive, the researcher welcomes them and asks questions that

prompt them to talk about a wide range of recent online experiences, such as “What

have you recently shopped for online?” or “Have you planned any trips recently?”

C

C

The researcher provides customers with a computer or mobile device that allows

them to browse the web while recording their online session and facial expressions.

(Of course, customers are informed in advance that their session will be recorded.)

74

Refining Design foR Business

C

C

The researcher invites the customers to go through a scenario they had mentioned

earlier that pertains to the business (e.g., booking a hotel room for an upcom-

ing stay). Customers are asked to speak aloud while using the computer to let the

researcher know what they’re thinking.

Observations

C

C

The researcher then sits back, pays close attention, and tries not to say anything. The

key is to leave customers alone while carefully watching how they try to reach their

goal—and what obstacles they face along the way—and paying close attention to

shifts in body language, facial expressions, and so on.

C

C

The researcher refrains from asking customers what they think about specific web-

sites or designs. For example, the researcher does not ask whether certain sites are

easy or helpful. Instead, the researcher focuses on what customers show through

their actions; the more customers are prompted, the more likely they’ll be to say or

do something based on what they think the researcher wants to hear. If customers

get confused and ask what to do, the researcher simply says something like, “Please

do whatever you would normally do.”

On prompting the customer

C

C

Some customers have a tendency to remain silent throughout the session. If this

is the case, the researcher will occasionally prompt them with a kind reminder to

vocalize their thoughts.

C

C

If, toward the end of the session, customers haven’t used the online business, the

researcher may prompt them by saying something like, “I’m interested in seeing you

do the same thing with a few specific websites.” The researcher will give the custom-

ers a list of sites that includes the business being studied as well as competitors not

yet visited.

C

C

The researcher should wait until the end of the session to ask any specific questions

about the customer’s behavior, and even then, they should avoid asking for any opin-

ions. For example, a researcher can ask customers to explain what they were think-

ing at a specific moment rather than asking if they thought the experience was good

or not. When listening to their answers, the researcher should be on the lookout for

a mismatch between what customers say and their observed actions. People tend to

jump around online from website to website very quickly and usually have difficulty

remembering all the actions they took online, let alone why they took them.

C

C

At the end of the session, the researcher should thank the customers for their time

and pay them the agreed-upon fee.

After the session, the researcher reviews the video to capture any findings that provide insights into the list of qualitative research questions. These insights, in turn, may lead to the formulation of new questions for further exploration in the iterative process.

ChapteR 4: QuALitAtive ReseARCh

75

Other research Methods

Although heuristic reviews and observational customer research are the most common

methods, there are many other types of qualitative customer research. This section out-

lines some that are popular among online businesses.

It should be noted that two methods that remain popular today—focus groups and

usability testing—do not figure in this section because they are not recommended.

Focus groups tend to place customers in the role of product designer by explicitly

asking them to give advice on design decisions, which is difficult for people to do

when such decisions are not their area of expertise. Additionally, focus group partici-

pants tend to try to say the “right thing” to please the researcher or they succumb

to groupthink.

Traditional usability testing asks customers to complete a predetermined list of online

tasks based on actions the business would like to see completed rather than allowing

customers to show the business their natural online behavior. Even though customers

may be good at figuring out how to perform a task when asked, doing so takes them

out of their typical frame of mind when interacting with websites.

Instead, the following are several tried-and-true methods for understanding customers

through qualitative research.

ethnographic studies

Ethnographic studies are a special form of observational research that take place where

the customer would normally interact with the company, such as at their home or

office, or at the location of the business itself, so that they are more likely to behave the way they normally would. Just like observational customer research, these sessions

are most effective when they are unguided and researchers focus on active watching

and listening.

Kenyon Rogers, Director of Digital Experiments for Marriott International, said that

his business regularly conducts ethnographic studies at select hotel properties. For

example, in 2013 the company piloted a program that allows guests to “use their smart-

phones to check into the property and open their room door without needing to inter-

act with a Marriott team member.” Rogers added that very soon, “guests will be able to

control their entire experience, including ordering room service, extending their stay,

ordering transportation, and booking meeting rooms through their smartphones.”

76

Refining Design foR Business

Look for the post-its:

pete Maher on Contextual Inquiry

pete Maher is the Co-Founder and Chief Operating

great intentions—trying to protect a user’s security—

Officer of the LUMa Institute, an educational com-

enact requirements that ultimately make it too diffi-

pany that equips people to accelerate innovation. he

cult for people to achieve their goals.

is also the co-author of Innovating for People: Hand-

So, when conducting contextual inquiry research,

book of Human-Centered Design Methods, which has

we’ll advise teams to look for the post-its or duct

been adopted by the U.S. Office of personnel Man-

tape: What are the clever workarounds users come

agement’s Innovation Lab as core training material

up with to make it possible for them to use the

for federal government employees. In this interview,

product? It’s only when we ask people to show us

he shares his thoughts on the importance of primary

how they actually use something that we’re able to

observational research.

uncover really ripe opportunities for innovation. and,

Your book presents the “contextual inquiry”

most important, close observation uncovers those

method, which is an ethnographic approach. Can

gaps between what people do and what people say

you describe it here?

they do.

I’ve got a deep affinity for contextual inquiry research

Would you tell us more about the gaps between

because it’s a great way to get to insights that can ulti-

what people do and what they say?

mately uncover opportunities for innovation. Contex-

Margaret Mead, the famed anthropologist, really

tual inquiry takes place where the participants would

said it best: “What people say, what people do, and

normally conduct their tasks. The interviewer asks

what they say they do are entirely different things.”

the participants to go about their tasks in a normal

anybody who has spent a lot of time doing research

way, observes their actions in an unobtrusive man-

knows this to be the case. It’s not because people are

ner, interjects questions at opportune moments, and

necessarily trying to lie or mislead. It turns out that

records the sessions for later analysis.

we, as people, are simply not wired to be able to really

Would you share an example?

articulate why we did that thing we just did.

When we don’t deeply understand the context of

For example, if you ask somebody to describe to

the user, we sometimes assume that we’re deliver-

you in detail how they completed a purchase on a

ing against their needs. For example, it turns out

website, you could imagine them describing the steps

digital products are incredibly difficult to use. From

that they took and all the different things that they

my days of doing research in the financial services

did along the way. But if you were to actually watch

industry, when we would go into people’s homes to

that scenario play out, you would see something very

understand how they engage with financial products,

different. So contextual inquiry research, especially in

we would commonly find things like post-its stuck

the digital context, allows us to observe how some-

to monitors. It’s no mystery that some of those post-

body’s moving through a digital experience to really

its contain usernames and passwords, or software

know what they just did and to try to understand why.

instructions, because a lot of times companies with

ChapteR 4: QuALitAtive ReseARCh

77

surveys

Businesses place surveys on their actual site or app, or email them to customers. Large

surveys can have statistically significant sample sizes, but the researcher must be on

the lookout for data not representative of the larger customer base due to self-selection bias. For example, not every customer wants to fill out a survey, and those who do may

have the strongest positive or negative opinions.

As with all forms of qualitative research, the more open-ended the survey, the better.

Surveys that ask customers about specific design decisions place the customer in the

awkward position of being asked to provide advice outside of their area of expertise.

Rather, understanding whether customers found their overall experience to be positive

or negative and providing an open-form field for customers to write about any aspect of

the experience they choose can often lead to actionable data.

Customer Panels

Customer panels are a subset of surveys: They typically consist of thousands of partici-

pants who have elected to give survey feedback on a regular basis. Panels may be run

by a company’s research team or by consulting firms on behalf of many businesses.

Like surveys, customer panels can provide statistically significant sample sizes, but

it’s important to understand the segment of participants being queried. For example,

although panels consisting entirely of self-selected users of one business might not

be representative of the entire population, they can give the business insight into the

behavior and opinions of their more loyal customers.

Eileen Krill, research manager at The Washington Post, oversees customer research for all of the business’s print and digital brands. A panel of about 7,000 customers is included in the many types of qualitative and quantitative customer research she oversees. Krill

will ask the panel “a wide range of closed and open-ended questions, depending on the

objectives of the survey,” including “satisfaction rating questions.” She points out that

“open-ended feedback is generally far more meaningful and actionable than the score

itself ” because it can help to “reveal the reasons behind the scores.”

One question she often asks the panel is “whether a new product or feature will

improve the customer’s impression of The Washington Post brand.” Although in most cases participants say such additions would have no impact, Krill still asks the question in case it provides an important insight. For example, she said, “We once tested the idea of starting an online dating service for Post readers, and that got a lot of people saying they would have a lower opinion of the company.” Krill noted that “I think that research was one of the key things that may have killed the idea.”

78

Refining Design foR Business

Diary studies

Diary studies consist of a business asking customers to take notes and regularly send

them back to the company, usually over an extended period of time. These studies may

ask participants to take notes only on a specific topic area, like their regular interactions with a new site or app, or they may be more general and simply ask customers

how they spent their day.

Google Search Lead Designer Jon Wiley shared an example of an ongoing diary study being

run by the company. Through a mobile app, the study regularly asks participants to reply to the question: “What is the last bit of information you needed to know?” The information can be related to any aspect of their life, not only the material they were looking for online. Wiley and his team then “look at the needs that people have in their lives” and try to answer the question, “Is there a way that we, as Google, can find a solution for them?”

Card-sorting

A technique used to gain insight into how to organize content, such as ordering and

grouping similar navigational links, card-sorting directly involves customers in the

design process. Customers are provided with a stack of cards containing information

and asked to perform a task, such as organizing them into logical categories. Although

it is typically performed with index cards or sticky notes, card-sorting can also be performed online, and there is no limit to the number of participants.

Caution should be observed, because this technique is more guided than the aforemen-

tioned qualitative methods and may place customers in the position of being asked to

act like a professional designer. However, if performed with minimal prompting, card-

sorting can provide designers with a rare opportunity to gain insight into how custom-

ers think about information architecture and content hierarchies.

Nate Bolt, design research manager at Facebook, said the company used cart-sorting as

one of many qualitative research methods to inform an ongoing redesign of users’ Face-

book News Feeds. After recruiting users, the Facebook Design Research team printed

out each user’s feed, up to the minute, on paper. Then, he explained, “they would cut out their feed stories, place them on a table, and group them into what they considered to

be like-minded categories. That helped us reprioritize the ways that stories are grouped and organized within people’s News Feeds in a real, human way. Obviously, this went

hand in hand with all other data, including the system (analytics) data.”

Feedback Forms

Feedback forms on websites, or email addresses for user feedback, can be a good source

of information. Users who provide open-ended feedback in this way often have timely

insights into customer frustrations, as well as positive comments on great experiences

with the website or the business’s team members.

ChapteR 4: QuALitAtive ReseARCh

79

John Williamson, the Senior Vice President and General Manager of Comcast.com,

explained that “One of the best sources for information I have is the ‘Website Feedback’

link” that appears on the bottom of each page on the site. Williamson noted, “I was in

banking in the mid-1990s, and I remember how excited we would get if we received a

letter from a customer. We really would.” Every day Comcast receives “hundreds of let-

ters” from customers through the feedback link, and because they are a “huge benefit”

that helps Williamson and his team understand their customers and their business, he

added, “I read every one of them.”

Acting on Qualitative Research

After the heuristic and other qualitative research sessions have been conducted,

researchers summarize the insights they have garnered, as well as any remaining ques-

tions about customer behavior. For each insight, they write down possible ways of veri-

fying the observations through analytics data, as well as any ideas for testing. This list will be used as a guide when crafting the Optimization Roadmap.

Case Study: Comcast

A leading provider of television, high-speed Internet, and digital voice services in the United States, Comcast uses qualitative and quantitative research to inform the iterative design process it applies to all aspects of its site.

Qualitative studies play a critical role in Comcast’s design process. Williamson noted

that in one redesign process during 2012, “We had over a thousand customers and pro-

spective customers involved before we tested online….We had ethnographic research,

customer panels, and online customer panels.” The Comcast team also “went into the

homes of over 20 customers and saw how they interacted with Comcast,” which pro-

vided “critical” experiences that analytics “data cannot replace.”

The following business case shows how insights gathered from qualitative studies trig-

gered a redesign of the company’s “TV Options” page, which customers use when pur-

chasing television service online. This section walks the reader through the redesign

process from beginning to end, showing how the qualitative study findings led to the

formulation of hypotheses and redesign options, and finally to the first tests and the

selection of highly effective new designs.

The goal of the “TV Options” page is to help customers choose their receiver type and

to select service upgrades, such as premium channels. However, qualitative studies sug-

gested three page elements were underperforming: receiver selection, premium channel

logos, and other add-on options further down on the page. The default version of the

page, with these three elements highlighted, appears in Figure 4.2.

80

Refining Design foR Business

Default

Figure 4.2 The default

page with three elements

highlighted for testing.

ChapteR 4: QuALitAtive ReseARCh

81

The qualitative study findings related to these three elements and the steps researchers took in response are summarized here.

Finding. Not all customers interacted with the first step of the process to learn more about upgrading to an HD or HD DVR receiver.

C

C

Hypothesis. Customers do not notice the option to upgrade a receiver.

C

C

Validate with analytics. Does the link to upgrade the receiver get very few clicks?

C

C

Test idea. Find out whether more customers would upgrade if more information

about the options to switch to an HD or HD DVR receiver appeared on the page

without requiring additional clicks.

Finding. All customers understood how to select and deselect premium channels, so this function seems user friendly. However, the website highlighted specific shows carried on these channels rather than the actual channels, and several customers commented that

they hadn’t heard of these shows. Example: “I never heard of Boss. I never saw Contagion.”

C

C

Hypothesis. Highlighting channel names rather than specific shows will clarify the process of adding premium channels.

C

C

Validate with analytics. Do premium channel selections vary based on the

featured shows?

C

C

Test idea. To appeal to a wider audience, feature prominent channel names

(HBO, Showtime, etc.) rather than specific shows.

Finding. Several customers commented that they didn’t know what was included in

the add-ons. Example: “What exactly is included in the Sports Package?”

C

C

Hypothesis. Customers don’t realize they can click on the toggle switches for more information. Because the switches are arranged in rows, followed by messaging for

each add-on, customers may be mistaking them for bullet points.

C

C

Validate with analytics. Do customers rarely click on these toggle switches?

C

C

Test Idea. Replace individual offers with a one-line message reading “More custom-ization options” to help customers understand that the image to the left is a toggle

switch that, when clicked, will reveal the details of the offers.

As noted earlier, researchers targeted receiver selection, premium channel logos, and add-on content for design testing. An alternate version of each element was built and placed in a multivariable test (MVT). A multivariable, or multivariate, test is a special type of A/B

test in which several page elements are tested across multiple recipes. Rather than simply testing two versions of the page—one with the original elements and one with the new

ones—multiple versions of the page are tested, each including a different combination of the elements. In addition to determining which version of the page has the highest performance, this type of test measures the success of each element.

82

Refining Design foR Business

The default and alternate versions of each element are shown in Figure 4.3. This figure also includes which version of each element won and which out of all three elements

most influenced the performance of the page:

C

C

Receiver section. The default version has a blue link to upgrade. The alternate version adds a prominent message to upgrade to an HD or HD DVR receiver.

C

C

Premium channel logos. The default version features a specific show for each premium channel. The alternate version features channel logos instead of shows.

C

C

Other add-ons. The default version places content behind four toggle switches. The alternate version places content behind a single toggle switch, which, when clicked,

expands to show all up-sell content at once.

The business goal was to increase the purchase conversion rate as well as the overall

revenue per visitor (RPV). As Figure 4.3 shows, the winning version of the receiver

selection element was not the new design but the original; however, the new versions

emerged as winners for both the premium channel element and the add-ons further

down on the page.

When Comcast integrated all three winning designs into the page and tested this

revamped page against the original one, it found the changes drove a 4.6 percent lift

in the purchase conversion rate and a 5.6 percent lift in RPV. The default and winning

versions of the page are pictured in Figure 4.4.

Because the impact of each element had been isolated, Comcast was also able to mea-

sure how much each one influenced the performance of the final version of the page:

The premium channel element was the most influential, accounting for over 60 percent

of the increase in RPV, indicating that, of the three elements tested, it was the original version of this feature that had presented the largest obstacle to customers completing

their goals.

The main recommendation was to push the winning version of the page to all visitors.

Additional proposals included running follow-up design tests of each variable to further simplify the user experience, starting with the premium channel element, because it

had emerged as the most important one overall.

ChapteR 4: QuALitAtive ReseARCh

83

Default elements

Alternate elements

WinneR

element #1

Default: Minimal upgrade

Alternate: More prominent hD or

messaging

DvR upgrade messaging

WinneR + MOst inFLuentiAL eLeMent

element #2

Default: specific show for each channel

Alternate: Channel name without shows listed

WinneR

element #3

Alternate:

All content hidden

Default: Three collapsed choices

behind one link;

when clicked,

all content expands

Figure 4.3 The default and alternate versions of each of the

three elements tested. This figure includes the winning version

of each element, as well as the most influential of the three

winning elements in terms of overall impact on the page.

84

Refining Design foR Business

Default

Winner

+5.6% lift in RPv

+4.6% lift in purchase conversion rate

Figure 4.4 The default version of the page next to the

redesigned page, which contains all the winning elements.

ChapteR 4: QuALitAtive ReseARCh

85

take a Step Back

Although qualitative research is a key part of the ongoing design process, it can also

play an invaluable role in helping to shape a company’s overall direction. Grounded in

the customer perspective, qualitative research helps businesspeople to shape long-term

strategies by keeping their finger on the pulse of an evolving marketplace, a practice

that helps them develop better ways of fulfilling their customers’ needs in the present

and in the future.

As businesspeople focus on the day-to-day activities related to delivering the same

products and services, it’s easy to lose sight of the long-term customer needs that are driving the demand for those same products—a phenomenon renowned marketing

savant Theodore Levitt called “marketing myopia.” Levitt argued that these companies

tend to be “product-oriented instead of customer-oriented,” and view their “marketing

effort” as a “necessary consequence of the product, not vice versa, as it should be.”

He cited the example of railroad companies: Once the titans of industry, they had

fallen on hard times by the 1960s, all because they thought of themselves as being

in the business of “railroads.” Had they instead seen themselves as being in the busi-

ness of “transportation,” they would have begun to produce cars, trucks, airplanes,

and even telephones—an expansion they were well positioned to tackle, given their

extensive resources.

Levitt may have been writing in the 1960s, but his observations remain true: All too

often, businesses focus on current successes instead of prioritizing their customers’

present—and future—needs. In recent years, however, some companies have grown to

recognize the importance of building for the future; a few examples of well-developed

marketing strategies, or long-term game plans for providing customer value, appear in

Table 4.1.

Each strategy conveys a straightforward understanding of why customers come to the

business for help that goes beyond the current core product and service offerings. For

example, Comcast’s strategy to focus on entertainment and technology doesn’t keep the

company tied to current revenue sources, such as cable lines or set-top boxes, because

these will likely be short-lived due to rapidly evolving technology; indeed, Comcast

has expanded its product offerings to allow customers to watch movies and television

shows from mobile devices even when offline. Similarly, IBM’s online strategy is focused on work-related tasks for business customers; as such, IBM has expanded its services

into cloud computing, “big data,” and social technologies.

All these examples illustrate the importance of preparing a pathway for the future—

a journey that, in many cases, begins with asking solid qualitative-research questions

and paying close attention to the answers.

86

Refining Design foR Business

table 4.1 Online Marketing Strategies

COMPAnY

OnLine MARketinG stRAteGY

ally Bank

“We help our customers to achieve their savings goals.” (andrew Switzer)

Comcast

“Comcast helps our customers to enrich their lives through entertainment

and technology.” (John Williamson)

Foursquare

“We help our customers to make the most out of where they are.” (Simon

Favreau-Lessard)

IBM

“IBM helps enable our customers to do their jobs better.” (phil Corbett)

LinkedIn

“We help our customers to connect with new opportunities.” (amy parnell)

petCarerx

“petCarerx helps to bring health and happiness to pet owners and their

pets.” (Blake Brossman)

Saks Fifth

“Our strategy is to inspire customer confidence and style with every Saks

avenue

shopping experience.” (roger Scholl and Matt Curtis)

WebMD

“We help our customers to find health information.” (rob Blakeley)

Up Next

This chapter provided an overview of qualitative research techniques, which offer

insights into customers’ goals and the challenges they face when trying to accomplish

those goals. These insights lead to test ideas that will be added to the Optimization

Roadmap, as well as inform the overall direction for a company.

The next chapter outlines quantitative research methods that companies use to verify

qualitative findings, as well as to identify which site areas need improvement the most

and what test ideas to prioritize as part of assembling the Optimization Roadmap.

references

1. “Marketing Myopia,” Harvard Business Review; Theodore Levitt (1960).

This page intentionally left blank

INDEX

promoting social links at log-off portal,

A

148–149

A/B testing. See also testing

American Express, 150–152

Adobe’s use of, 143–145

business and customers’ goals for, 150

Ally Bank’s use of, 147–149

designing multiple product displays,

best practices for, 114

151–152

Caesars’ booking module, 154–156

offering customer incentives, 151

Comcast’s, 160–161, 162

optimization program for, 150

defined, 29

testing impact on business strategies,

Dell’s use of, 168–169, 170–171

150–151

email design for Dollar Thrifty, 173–176

analysis paralysis, 88

Foursquare’s, 191, 193, 194–195

AOV (average order value), 92, 166, 267

Hightail’s homepage test using, 211

assembling wireframes, 119–120

LinkedIn’s, 223–225

auto-focusing feature, 246

Marriott International, 227–230

auto-optimizing tests, 109, 116–117

multivariable tests in, 81

autocomplete feature, 194–195, 209

Obama for America, 231, 232, 233–234,

average order value (AOV), 92, 166, 267

236–248, 249–250

Axelrod, David, 234

PetCareRx messaging and banner, 252–257

product storage limits, 211–212

redesigned customer loyalty pages, 157

B

banners

segmentation and, 235

A/B testing of size, messaging, and images

specifying in methodology, 109

for, 253–255

T-Mobile phone payment options, 265–267

ignored by customers, 69

Washington Post’s , 271–277

best practices

address fields, 216

customer service, 121–122

Adobe, 143–145

Foursquare’s, 193

business and customers’ goals for, 143

Hightail’s, 211, 212

use of A/B testing, 143–145

Saks Fifth Avenue’s, 258

advertising banners, 69

Bezos, Jeff, 11, 268, 271

Ahmed, Zimran, 177–184

Blakeley, Rob, 66, 86

algorithm for News Feed displays, 187–188

blue-sky approach, 120

Ally Bank, 146–149

Bolt, Nate, 34, 35, 78, 185–190

balancing creativity with research, 146

borders of wireframes, 119

business and customers’ goals for, 146

bounce rate, 92

homepage testing by, 147–148

280

Refining Design foR Business

Bowman, Doug, 200

Capital Business, 268

brainstorming test ideas, 103–104, 105

card-sorting, 78, 188

brand styles, 234–235

case studies. See also specific businesses

Brin, Sergey, 204

by name

Brossman, Blake, 47, 62, 86

about, 141–142

business and customers’ goals. See specific

Adobe, 143–145

case studies

Ally Bank, 146–149

business plans, 104

American Express, 150–152

businesses. See also case studies

Caesars Entertainment Corporation, 120,

becoming optimization organizations,

153–157

22–23

Comcast, 79–84

cultural benefits of optimization, 31

criteria applied to, 10

evangelizing optimization within, 271

Dell, 166–172

existing to help customers, 8

Dollar Thrifty Automotive Group, 172–176

fear of data-driven design, 25, 31–33, 35

Electronic Arts, 177–184

hiring optimization team members,

Foursquare, 191–195

137–140

Hightail, 210–212

making decisions on bad data, 26–27

IBM, 213–219

online presence for, 3–4

Lenovo, 93–101, 102–103

pairing data with design, 34

Obama for America, 20–21

realigning to customer goals, 9

PetCareRx, 47–60, 251–257

role of design in, 4–5

Saks Fifth Avenue, 258–261

shaping market strategies, 85–86

Washington Post, The, 11–20

software as a service, 210–211

chat pop-ups, 171

starting and scaling optimization programs,

Chertudi, Mike, 143–145

160, 173, 223

classified job listings, 273–276

traditional design processes in, 26

click-through rate (CTR), 19–20, 92, 192–193

understanding customer goals, 8–9

Clooney, George, 243–244

using surveys, 77

Club Pogo, 177, 178

buttons, 244, 249–250

color

Donate page button, 249–250

C

Google’s research on, 35, 200–201

Caesars Entertainment Corporation, 120,

pairing text and background, 33

153–157

testing form field background, 246

business and customers’ goals for, 153

Comcast, 158–165

optimization programs for, 154

business and customers’ goals for, 158

redesign of customer loyalty pages, 157

global navigation testing, 161, 162

testing semi-annual booking module,

mobile product page design for, 160–161

154–156

online marketing strategies, 158–159

call-to-action elements. See CTA elements

optimization strategies of, 159–160

Campbell, Emily, 166–171

personalizing self-service messaging,

164–165

inDex

281

qualitative research used by, 79–84

Hightail’s removal of, 211

testing buyflow customer information,

testing IBM link for, 218

161–163

CTR (click-through rate), 19–20, 92, 192–193

communicating. See also email

Curtis, Matt, 86, 258–261

during design comp phase, 122–123

customer-centric design. See also customers;

Foursquare design success, 192–193

user experience

with funnel reports, 98–99

Dell’s use of, 166–167

optimization results, 47, 125–126, 214

developing, 6–7, 178

PetCareRx’s optimization results, 61–62

framework for, 7–8

sharing Caesars test results, 154

online use of, 142

ways of, 126–127

customer loyalty programs

consistency

American Express testing for, 151

data, 114, 125

Caesars Entertainment Corporation’s,

design, 122

153–154, 157

contextual inquiry method, 76

Club Pogo, 177, 178

conversion

Marriott Rewards program, 226, 227

daisy-chain, 235

promotional messaging for Pet Points,

number of visits before, 97

256, 257

conversion rates

targeting promotions to visitors, 228–229

Caesars’ redesign increasing, 157

testing Caesars’ redesigned pages for, 157

Dell’s security verification and, 167

customer service, 121–122, 161–163

evaluating, 97

customers. See also customer loyalty

programs; users

finding products with highest, 96

audiences of Washington Post, 268–269

measuring, 91

educating about business, 161–163, 265–267

plotting in funnel reports, 98–99

helpfulness to, 8

T-Mobile phone payment options, 267

identifying goals of, 66–67

coordinated optimization, 130, 131

interacting with business designs, 4–5

Corbett, Phil, 86, 213–219

meeting expectations of, 122

corporate reliability, 167

observing, 72–74

cost effectiveness. See also RPV

offering products for subscribers and

Comcast’s ROI, 159

guests, 177–184

economy in design testing, 30–31

online behavior of, 68–69

monthly recurring revenue, 210

prompting in test sessions, 74

T-Mobile’s ROI, 263

realigning business with goals of, 9, 211–212

creativity

recruiting for research, 73

decision making using data and, 185–186

researching goals of, 8–9

design testing’s impact on, 31–33

responding to annoyances of, 171

effects of pixel politics on, 27–28

self-service programs for, 159

credit card input errors, 237–238, 240–241

setting up segments representing, 117–118

CTA (call-to-action) elements

testing new products and services with, 9–10

Comcast’s testing for, 160–161

within and outside organization, 226

Dell’s simplification of, 168

customers’ goals. See specific case studies

282

Refining Design foR Business

D

industry trends in, 223

LinkedIn’s approach to, 221–222

Daily Information Needs (DIN), 196–198

Marriott panel, 227–228

daisy-chain conversions, 235

measuring success of, 192

data. See also metrics

meeting customer expectations, 122

algorithm for News Feed displays, 187–188

mitigating risk of new, 30

analysis of Optimization Results, 46

redesigning classified job listings, 273–276

analysis paralysis, 88

simplicity in, 120, 166–167

collecting user’s experiential, 196, 197–198

smallest screens first, 120–121

decision making using, 159, 185–186

testing multiple designs, 28–31

ensuring consistency of, 114

traditional process of, 26

fluctuation of, 93–94

Washington Post’s calendar redesign, 276–277

helps in understanding analytic, 66

design comps

insights and impact analysis for, 89–90

defined, 45–46

learning details behind, 66

developing PetCareRx, 55–60

Lenovo’s use of analytic, 93–101

tips for creating, 122–123

making decisions on bad, 26–27

translating wireframes into, 46

pairing with design, 34

designers

targeting promotions based on visitor,

228–229

balancing measurements with talent of, 146

use in design decisions, 29–30, 32

fears about design testing, 25, 31–33, 35

using online success metrics, 90–93

Google’s use of design teams, 203–204

Dell, 166–171

response to Google’s color testing, 200–201

business and customers’ goals for, 166

diary studies, 78

online marketing strategy for, 166–167

DIN (Daily Information Needs), 196–198

optimization programs for, 167

Dinner with Barack page, 20–21, 243–244

personalizing homepage experience,

documentation

170–171

developing Executive Summary, 51–52

removing chat pop-ups, 171

how to create, 43–44

segmentation models for, 166

importance of, 42

testing coupons and shopping carts,

sharing online spreadsheets, 236

168–169

sharing review results, 70–72

tracking results for, 167

dog fooding, 199

design

Dollar Thrifty Automotive Group, 172–176

artistry in, 33

business and customers’ goals for, 172

consistency in, 122

email opt-out designs for, 173–174

creativity vs. testing in, 31–33

optimization program for, 172–173

developing customer-centric, 6–8, 142,

testing email hero offers, 174–176

166–167, 178

donations. See also fundraising

effects of pixel politics on, 27–28

saving payment information page for,

Facebook’s approach to, 185–186

241–242

flat vs. skeuomorphic, 201

testing Donate page, 238–240, 244–248,

importance to businesses, 4–5

249–250

inDex

283

E

innovation by teams at, 186

Electronic Arts, 177–184

redesign of News Feed, 186–190

business goals for, 177

release of News Feed layout, 189

games marketed by, 177–178

research methodologies used by, 186–187

product testing by, 178

team role in innovation, 186

Word Whomp testing by, 179–184

failing faster, 30

element discovery tests, 115

Favreau-Lessard, Simon, 86, 191–195

email

feedback

contests collecting addresses, 20–21

Google’s use of customer, 199

fundraising strategies for, 232

using customer panels, 77, 269

LinkedIn’s mobile template testing, 223–224

website forms for, 78–79

opt-out designs for, 173–174

field studies at Google, 198

reinforcing content on landing page, 244,

fields

245

address, 216

segmentation in, 235

auto-focusing on first name, 246

sending to targeted customers, 172–173,

finding entry errors for, 232–233

174–176

marking required, 238, 239

updating lists regularly, 127

reducing credit card input errors, 237–238,

embedded optimization culture, 130, 131

240–241

errors

T-Mobile testing of input, 263–265

detecting common entry, 232–233

testing background color of, 246

effects on site metrics, 124

flat design, 201

marking required fields to reduce, 238, 239

flipping a switch, 27

reducing credit card input, 237–238,

focus groups, 75

240–241

fold lines, 119, 121

simplifying form to reduce, 238–240

Ford, Henry, 200

Type I/Type II, 109–110

Foursquare, 191–195

ethnographic studies, 75–76

best practices for testing, 193

executive sponsor, 135, 137

business and customers’ goals for, 191

Executive Summary, 51–52, 61–62

innovation at, 191–192

experiential business review, 69

measuring design success at, 192–193

experiential competitive reviews, 69–72

testing autocomplete for search results,

experiments. See testing

194–195

Express, 268

free services

Hightail’s paid and, 210

F

Word Whomp testing of, 182–184

free shipping offers, 256, 257, 261

Facebook, 34, 78, 185–190

Fuhriman, Jeff, 143–145

algorithm for News Feed displays, 187–188

full-factorial multivariate test, 115

Bolt’s role with customers at, 185

fundraising

card sorting by, 188

auto-focusing on first name field for, 246

data and creativity in design of, 185–186

daisy-chain conversions for, 235

284

Refining Design foR Business

fundraising ( continued)

H

designing donation pages, 232–233

headlines, 121, 215–216

form field background color tests, 246, 247

hero offers and images, 174–176, 230

reducing credit card errors, 237–238,

Hertz Global Holdings, Inc., 172

240–241

heuristic reviews. See user experience

reinforcing email content on landing page,

244, 245

Hightail, 210–212

removing dollar signs, 246

business and customers’ goals for, 210

removing donate buttons from site, 244

homepage testing by, 211

strategies for, 231–232

success metrics for, 210–211

testing video for, 242, 243

testing product storage limits, 211–212

funnel report, 98–99

hiring team members, 137–140

homepage

G

Ally Bank’s testing of, 147–148

Dell’s testing of, 170–171

games, 177–184

designing new versions of, 6–7

geo-targeting, 95

Hightail’s testing of, 211

global navigation testing, 161, 162

Marriott’s testing for, 227–229

global pre-login page, 215–216

PetCareRx message testing for, 252–253

Goff, Teddy, 236

prioritized test list for, 51

Going Out Guide redesign test, 276–277

hypotheses, 108, 109

Google, 196–209

approach to innovation, 196–198

I

collecting experiential data, 196, 197–198

developing query information sets, 204–207

IBM, 213–219

Instant, 209

blog about optimization results, 214

Knowledge Graph, 202, 204, 205, 207

business and customers’ goals for, 213

qualitative research used by, 198–199, 206

MVT testing by, 215–219

research on color, 35, 200–201

optimization program of, 213–214

Search, 34, 201–202

success metrics for, 214

search box modifications, 207–209

Illustrator product page, 143–145

sharing results via Google Documents, 236

images. See also case studies

success metrics tracked by, 201–203

A/B testing of links to photo tour, 230

teams within, 203–204

A/B testing of products and, 151–152

Voice Search, 200, 202

PetCareRx A/B testing of banner, 253–255

groupthink, 75

testing alternate, 21

guests

implementing test design, 123

logging in with IBM ID, 216–219

incident optimization, 130, 131

offering products for subscribers and,

Innovating for People (Maher), 76

177–184

innovation

guided editing flow, 224–225

creating new user experiences, 199–200

encouraging with design testing, 30

inDex

285

finding opportunities for, 76

KPIs (key performance indicators), 153

Foursquare’s approach to, 191–192

Krill, Eileen, 11, 77, 268–277

Google’s approach to, 196–198

LinkedIn’s approach to, 221

L

role of Facebook teams in, 186

Land, Edwin, 185

separating fads from, 104

Lau, Thomas, 150–151

insights and impact method

layouts

about, 89–90

best practices for, 120–121

evaluating Lenovo’s site metrics with, 94,

95–98, 101

IBM’s MVT testing of, 215–216

Lenovo’s use of, 93–101

PetCareRx test, 56–60

interviewing job candidates, 138–139

redesign of Facebook’s News Feed, 186–190

Iterative Optimization Methodology. See also

legal approvals, 123

specific methodology stages

Leibrock, Jane, 187, 188

about, 7–8

Lenovo, 93–101

core principles of, 8

path reports for, 99–101

documentation of, 42, 43–44

prioritizing test areas for, 102–103

illustrated, 43

using insights and impact method, 94,

Optimization Launch, 42, 46

95–98, 101

Optimization Plan, 42, 45–46

Levitt, Theodore, 85

Optimization Results, 42, 46–47

liberal arts backgrounds, 137–138

Optimization Roadmap, 42, 44–45

lifetime revenue (LTV), 210

organizational use of, 22–23

LinkedIn, 34, 220–230

realigning business with customer goals, 9

approach to design, 221

researching customer goals in, 8–9

business and customers’ goals for, 220

stages in, 42–43

challenges for, 221

starting from helpful point of view, 8

good user experience for, 220–221

trying new products and services, 9–10

innovation at, 221

iterative tests

measuring design success for, 222

confirming results with, 226–227

optimization teams for, 223

Google’s use of, 206–297

role of research at, 221–222

numbering, 106

segmentation, targeting, and

personalization for, 222–223

Obama for America’s use of, 244–245,

249–250

links, 216, 217, 218

developing QA, 125

J

MVTs for help, 216, 217, 218

promoting social, 148–149

job descriptions, 138, 139

Saks Fifth Avenue’s testing of, 259–261

testing appearance of IBM, 215, 218

K

testing placement of photo tour, 230

Kahle, Chris, 119–120, 153–154

website feedback, 79

Knowledge Graph, 202, 204, 205, 207

log-off portal testing, 148–149

286

Refining Design foR Business

login pages

methodology for using segments and,

IBM global pre-login page, 215–216

117–118

target testing for Obama for America,

Obama for America’s, 235

242–243

online success, 90–93

Washington Post, 271–273

ranking pages by key, 94–95

LTV (lifetime revenue), 210

tracked by Google, 201–203

LUMA Institute, 76

using online, 90–93

minimum recommended traffic levels,

M

110–112

mobile devices. See also T-Mobile

Maher, Pete, 76

Comcast product page for, 160–161

marketing. See online marketing strategies

designing first for, 120–121

marketing myopia, 85

LinkedIn’s email template for, 223–224

Marriott International, 226–230

single click donations from, 241–242

business and customers’ goals for, 226

MRR (monthly recurring revenue), 210

ethnographic studies used by, 75

multivariable tests. See MVTs

homepage A/B testing, 227–229

MVTs (multivariable tests), 81

metrics for, 226–227

IBM’s use of, 215–219

testing hero image, 230

prepaid activation field testing, 263–265

Martin, Sandy, 172–176

specifying in test methodology, 109

maturity model checklist, 130–132

types of, 114–115

Mead, Margaret, 76

messaging

N

A/B testing of PetCareRx, 252–257

best practices for, 121

naming recipes, 119

Comcast’s Social Security, 161–163

navigation testing

personalizing self-service, 164–165

Comcast global, 161, 162

working with political, 234–235

LinkedIn’s, 222

methodologies. See Iterative Optimization

Saks Fifth Avenue, 259–261

Methodology; test methodologies

New User Experience (NUX), 187

metrics

New York Times, The, 35

changing, 113

News Feed layouts, 186–190

conducting QA on important, 124

numbering iterative tests, 106

consistency across recipes, 125

NUX (New User Experience), 187

Dell’s use of, 166

developing funnel reports for, 98–99

O

fluctuation over time, 93–94

Obama, Barack, 20–21

Foursquare’s, 192–193

Obama for America, 231–250

Hightail’s testing, 210

A/B testing for, 231, 232, 233–234, 236–248,

IBM’s, 214

249–250

Marriott’s, 226–227

adding presidential quote to, 246, 248

measuring LinkedIn design success, 222

brand styles and political messaging of,

234–235

inDex

287

customer research by, 232

Caesars Entertainment as, 154

Dinner with Barack page, 20–21, 243–244

Comcast strategies as, 159–160

Donate page testing for, 244–247

creating team for, 132–133

Kyle Rush’s role with, 231

cultural benefits of, 31

metrics used for, 235

Dell as, 167

saving payment information page for,

developing business culture for, 22–23

241–242

Dollar Thrifty as, 172–173

segmentation, targeting and

hiring team members, 137–140

personalization for, 235–236

IBM as, 213–214

site goals for, 231–232

Lenovo’s development as, 93–101

targeting login link testing, 242–243

maturity model checklist for, 130–132

testing platform speed for, 236–237

starting and scaling programs in, 160, 173, 223

video testing fork, 242, 243

steps for applying optimization method, 130

observational customer research, 72–74

T-Mobile as, 262–263

online marketing strategies

team roles and responsibilities in, 133–135

banner blindness, 69

testing design as, 6

Comcast’s, 158–159

Optimization Plan. See also design comps;

Dell’s, 166–167

wireframes

marketing myopia, 85

about, 42, 45

marketing online games, 177–178

components of, 45–46, 108

research shaping, 85–86

developing for PetCareRx, 51–60

online presence

roles and responsibilities for, 134

design and, 4–5

test methodologies in, 108–117

development of Washington Post’s, 11

optimization results

importance of, 3–4

blogs for communicating, 214

testing products and services with, 9–10

Caesars Entertainment, 154

online success metrics, 90–93

communicating, 47, 125–126

Optimization Launch

Dell’s tracking of, 167

about, 42, 46

IBM’s blog about, 214

implementation, 123

Obama for America’s tracking of, 236

PetCareRx, 55

PetCareRx’s, 61–62

QA process during, 124–125

roles and responsibilities for, 134

releasing News Feed layout, 189

Optimization Results stage

roles and responsibilities for, 134

about, 42, 46

optimization lead, 135–136, 137, 139

analysis of, 46

optimization methodology. See Iterative

communicating results, 47, 125–126

Optimization Methodology

Optimization Roadmap

optimization organization, 133–137. See also

about, 42, 44–45

teams; and specific case studies

assembling, 102–106

Ally Bank as, 146

PetCareRx, 47–51

American Express as, 150–151

roles and responsibilities for, 134

balancing existing and products in, 199–200

288

Refining Design foR Business

P

brainstorming on areas for testing,

103–104, 105

PACO (Personal Analytics Companion), 196

homepage test list of, 51

Page, Larry, 202, 204

setting design testing, 47–48

page views, 90, 92, 94

products

Parnell, Amy, 34, 96, 220–230

calls-to-action on mobile pages for, 160–161

partial factorial multivariate testing, 115

converting innovative ideas into, 198–199

path reports, 99–101

customer-centric design of, 178

payments

designing page with display for multiple,

options for T-Mobile phone, 265–267

151–152

providing security information during,

developing innovative Google, 198–200

161–163

free shipping offers for, 256, 257, 261

saving information for, 241–242

homepage display of Ally Bank, 147–148

Personal Analytics Companion (PACO), 196

marketing games, 177–178

personalization

simplifying Illustrator page design, 143–145

Dollar Thrifty’s use of, 172–173

testing Hightail storage limits, 211–212

effect on LinkedIn design, 222–223

Washington Post, 268, 270–271

Obama for America’s use of, 235–236

prompting customers in test sessions, 74

personalizing homepage experience,

170–171

self-service messaging, 164–165

Q

PetCareRx, 251–257

QA (quality assurance), 124–125

about, 47

qualitative research, 65–86

business and customers’ goals for, 251–252

about, 9

design comps for, 55–60

acting on, 79

developing Optimization Plan, 51–60

benefits of, 66–68

failed expansion ideas for, 252

card-sorting, 78

finding customers for, 252

customer panels, 77

illustrated, 48, 49, 52–54

diary studies, 78

messaging testing by, 252–257

ethnographic studies, 75–76

Optimization Launch for, 55

experiential business review, 69

Optimization Roadmap for, 47–51

experiential competitive reviews, 69–72

origins of, 252

feedback forms, 78–79

RPV lift results for, 61

Google’s use of, 198–199, 206

updated Executive Summary for, 62

identifying customer goals with, 66–67

pixel politics, 27–28

methods for, 75–79

Pizzuto, Ryan, 262–267

Obama for America’s use of, 232–233

planned optimization, 130, 131

observational customer research, 72–74

platform speed testing, 236–237

questions for, 67–68

price testing, 121–122

shaping marketing strategies with, 85–86

priorities

surveys, 77

basing on value and difficulty, 104–105

testing design ideas, 32

inDex

289

used by Comcast, 79–84

revenue and orders metrics, 90

user experience reviews, 68–72

revenue per thousand impressions (RPM), 192

Washington Post’s use of, 269–270

Revenue Per Visitor. See RPV

quantitative research, 87–101

Rogers, Kenyon, 75, 138, 226–230

about, 8–9

ROI (return on investment)

benefits of, 88

Comcast’s optimization program and, 159

Google’s use of, 199

T-Mobile’s testing and, 263

insights and impact method for, 89–90

roles and responsibilities

Obama for America’s use of, 232–233

Chris Kahle’s, 153

online success metrics, 90–93

design roles at Google, 203–204

questions for, 88–89

Eileen Krill’s, 268

queries

Kyle Rush’s, 231

developing sets of information for, 204–207

Nate Bolt’s, 185

Google metrics for, 202

Simon Favreau-Lessard’s, 191

recipes for refining Foursquare, 195

team, 133–135

questions

room bookings, 226–230

answering with quantitative research,

Root, The, 268

88–89

RPM (revenue per thousand impressions), 192

identifying priorities, 67–68

RPV (Revenue Per Visitor)

Quick Donate, 241

Adobe designs for increasing, 143–145

calculating, 90–91, 154–156

R

Comcast lifts in, 82, 84

Rathee, Manik, 234

defined, 51

recipes. See also specific case studies

Dell’s conversion rates and lifts in, 168–169,

170–171

naming, 119

increases for PetCareRx, 61

number tested, 109

Washington Post’s use of, 276

using consistent metric across, 125

Rush, Kyle, 20, 21, 231–250

Washington Post design, 12–19, 20

recruiting customers for research, 73

research. See also qualitative research;

S

quantitative research

Saks Fifth Avenue, 258–261

conducted by Washington Post, 269–271

best practices at, 258

done by Obama for America, 232

business and customers’ goals for, 258

Facebook’s methodologies for, 186–187

free shipping threshold testing, 261

Google’s color, 35, 200–201

left-hand navigation link testing, 259–261

Google’s DIN study, 196–198

measuring test success for, 258

recruiting customers for, 73

scaling

role in LinkedIn’s design, 221–222

Martin’s advice on, 173

results. See Optimization Results stage;

Parnell’s advice on, 223

optimization results; search results

user and A/B testing programs, 233–234

results library, 126

Williamson’s suggestions on, 160

return on investment. See ROI

290

Refining Design foR Business

Schmidt, Eric, 204

redesigning shopping carts for, 168–169

Scholl, Roger, 86, 258–261

splitting one form into four steps, 238–240

search results

skeuomorphic design, 201

developing querying sets for, 204–207

Slate, 268

Google search box modifications, 207–209

Smarter Planet website (IBM), 214

measuring effectiveness of, 192–193

social networking, 11–20, 148–149

recipes for query refinement, 195

software as a service (SaaS) business,

testing autocomplete feature for, 194–195

210–211

using Google Voice Search, 200, 202

strategic alignment, 130, 131

section types, 92

success metrics. See metrics

segmentation

surveys

Dell’s model for, 166

qualitative research using, 77

Dollar Thrifty Automotive Group’s use of,

Washington Post’s online user, 269, 271

172–173

Switzer, Andrew, 86, 146–149

effect on LinkedIn design, 222–223

examples of, 118

T

Obama for America’s use of, 235–236

T-Mobile, 262–267

setting up, 117–118

business and customers’ goals for, 262

testing pre- and post-game pages for user,

179–184

optimization programs within, 262–263

self-service

prepaid activation field testing, 263–265

Comcast’s philosophy on, 159

ROI for testing, 263

personalizing messaging for, 164–165

Tai, Linda, 210–212

senior hires, 138, 139

targeting

services. See also products

A/B testing on login links, 242–243

self-service programs, 159, 164–165

Dollar Thrifty customers, 172–173

software as a service businesses, 210–211

effect on LinkedIn design, 222–223

testing free, 182–184, 210

geo-targeting, 95

shopping carts

Obama for America’s use of, 235–236

Comcast’s testing for mobile, 160–161

promotions based on visitor data, 228–229

Dell’s A/B testing of, 168–169

search results for timeliness, 193

evaluating paths to, 99–101

testing, 116

PetCareRx messaging tests for, 255–257

teams

removing chat pop-ups from, 171

allocating member time in, 136–137

significance level

brainstorming test ideas, 103–104, 105

defined, 109–110

creating core optimization, 132–133

getting A/B test results quickly, 234

developing wireframes, 120–122

of Type I/Type II error occurrences,

executive sponsor for, 135, 137

110–112

Google, 203–204

simplicity

hiring members of, 137–140

in design, 120, 166–167

interviewing candidates for, 138, 140

developing News Feed redesign, 188–190

LinkedIn optimization, 223

inDex

291

optimization lead, 135–136, 137, 139

form field background color, 246, 247

prioritizing test areas, 102–103

Google’s use of, 35

regular meetings for, 127

hero offers and images, 174–176, 230

role in Facebook innovation, 186

hypothesis development for, 11–12

roles and responsibilities for, 133–135

IBM link for CTA, 218

senior hires for, 138, 139

keeping track of Caesars’, 154

setting testing priorities, 104–105

multiple design versions, 6

starting and scaling new programs, 160,

new products and services online, 9–10

173, 223

numbering tests in, 106

technical lead, 136, 137

platform speed, 236–237

technical lead, 136, 137

priorities for, 47–48, 102–103

test methodologies, 108–117. See also

purpose of, 113

A/B testing; Iterative Optimization

QA process during, 124–125

Methodology; MVTs

quantitative questions shaping, 88–89

components of, 108–109

roadmap for PetCareRx, 47–51

duration of testing, 109

Saks Fifth Avenue, 258, 259–261

evaluating effect of traffic levels on,

110–113

selected web browsers, 125

evaluating Type I/Type II errors, 109–110

site’s query refinement, 195

Facebook’s research, 186–187

Social Security messaging, 161–163

insights and impact method, 89–90

types of, 109

number of recipes in, 109–113

using data in, 29–30

purpose and effectiveness of testing, 113

Washington Post layouts, 11–20

used in qualitative research, 75–79

Thomas, Scott, 235

test objectives, 108

toolbox options, 11–19

test types

TOS (time on site), 93

A/B testing, 29, 81, 109, 114

touch devices, 120–121, 200

about, 114

traffic

auto-optimizing tests, 109, 116–117

estimating minimum visitors per design,

110–112

multi-variable testing, 114–115

methodologies for low volume site, 113

targeting and personalization tests, 116

trust

testing. See also test methodologies

building customer’s, 121

autocomplete feature, 194–195

testing messaging for, 252–253

benefits of, 31

two-sided Z-test, 112

brainstorming ideas for, 103–104, 105

Type I/Type II errors, 109–110

Comcast’s use of, 79–84, 159

courage required for, 262

creativity in iterative, 32

U

defined, 29

usability testing shortcomings, 75

designing user, 234

user experience

duration of, 109, 234

business responses to, 72

fears about, 25, 31–33, 35

collecting data about, 196, 197–198

292

Refining Design foR Business

user experience ( continued)

W

creating new, 199–200

Washington Post, The, 11–20, 268–277

defining good, 220–221

audiences of, 268–269

documenting, 70–72

customer panels by, 77, 269

experiential business and competitive

default template for, 12

reviews of, 69

design recipes for, 12–19, 20

Google engineers involved in, 204

developing online business model, 11

LinkedIn’s focus on, 223

developing test hypothesis, 11–12

reviews of, 68–72

Krill’s role at, 268

user profiles

primary products of, 268

monitoring use experience of, 222

product testing for, 270–271

testing guided editing of, 220, 224–225

research conducted by, 269–270

user test labs

results of testing, 19–20

Google, 206

sign-in and registration page testing,

Washington Post, 269, 270, 271

271–273

users. See also user experience; visitors

user test labs, 269, 270, 271

customers vs., 185

web browsers, 125

don’t focus on advanced, 122

web sites. See also homepage; and specific

Google field studies with, 198

case studies

Google’s collection of experiential data

art vs. function of, 33

from, 196, 197–198

detecting common entry errors, 232–233

live interviews of, 269–270

determining country from IP address, 95

query refinement for, 195

educating customers about business, 161–

testing experience of new, 187

163, 265–267

testing pre- and post-game pages for

effect of pixel politics on design, 27–28

segmented, 179–184

experiential business review of, 69

testing sites with volunteer, 234

experiential competitive reviews of, 69–72

feedback links on, 79

V

flipping switch on, 27

variables, 114–115

fluctuation of key metrics, 93–94

video testing, 242, 243

full-site redesigns of, 263

visitors. See also RPV

internal optimization, 214

determining entry criteria for, 109

listing page views for, 94–95

registering site, 271–273

online customer behavior on, 68–69

targeting promotions to, 228–229

online success metrics for, 90–93

tracking, 90

page types for, 92

visits

prioritizing areas for testing, 102–103

calculating number before converting, 97

redesigning TV Options page, 79–84

measuring time of, 90

reviewing, 68–72

Voice Search, 200, 202

setting testing priorities for, 47–48

inDex

293

template analytics for, 92

defined, 45

testing platform speed for, 236–237

developing for PetCareRx, 52–55

tracking page views, 90, 92, 94

PetCareRx, 52–55

WebMD, 66

structure of, 119

Webster, Steven, 134–135

tips for creating, 120–122

white space, 121

translating into design comp, 46

Wiley, Jon, 34, 35, 78, 196–209

Word Whomp testing, 179–184

Williamson, John, 79, 86, 158–165

wireframes

Y

assembling, 119–120

YouSendIt, 210

candidates’ skills in, 140

Document Outline

- Contents

- Introduction

- The Age of Optimization

- How This Book Is Structured

- CHAPTER 4 Qualitative Research

- Benefits of Qualitative Research

- User Experience Reviews

- Observational Customer Research

- Other Research Methods

- Case Study: Comcast

- Take a Step Back

- Up Next

- References

- Index

- A

- B

- C

- D

- E

- F

- G

- H

- I

- J

- K

- L

- M

- N

- O

- P

- Q

- R

- S

- T

- U

- V

- W

- Y

Table of Contents

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

How This Book Is Structured . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Part 1 creating engaging customer experiences

chaPter 4 Qualitative Research . . . . . . . . . . . . . . . . . . . . . .

Benefits of Qualitative Research . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

User Experience Reviews . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Observational Customer Research . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Other Research Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Case Study: Comcast . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Take a Step Back . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Up Next . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Introduction

Chapter 4

Case Study: Comcast

INDEX

, 81

s, 69

g, 78, 188

s, 77

76

s, 74

73

s, 70–72

, 67–68

, 82, 84

C

D

E

G

J

M

P

R

T

U

V

Y